fix: Fix Question Service seed-batch bugs and edge cases #35

xGladiate · 2025-10-12T16:14:02Z

Description:

This PR fixes the following issues with the question syncing process:

Only leetcode DB sync is completed when it returns
The cursor was updated to the nearest 200 questions instead of the last sync point.
Seed cursor only validates sync to leetcode-service instead of question-service.

Changes Made

Health Check for Question Service done before fetching leetcode questions into the leetcode-service.questions DB
The next sync now starts from the last successful sync point.
Database operations are strictly limited to inserts—no updates if the question already exists in the database. (it was previously not explicitly guarded)

Copilot

Pull Request Overview

This PR fixes synchronization issues in the LeetCode question seeding process by improving cursor management and database operations. It removes the "done" state tracking in favor of more robust pagination handling and optimizes the database schema for better performance.

Replaces "done" state with total-based cursor management to prevent re-fetching completed data
Changes from $set to $setOnInsert to prevent overwriting existing question data
Optimizes database indexes by adding unique constraint on titleSlug and improving compound indexes

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
backend-services/leetcode-backend-service/src/leetcode/seedBatch.ts	Removes done state tracking, improves cursor pagination logic, and changes upsert strategy to preserve existing data
backend-services/leetcode-backend-service/src/db/model/question.ts	Adds unique index on titleSlug and optimizes compound indexes for better query performance

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

backend-services/leetcode-backend-service/src/leetcode/seedBatch.ts

Copilot · 2025-10-12T16:14:42Z

backend-services/leetcode-backend-service/src/leetcode/seedBatch.ts

+        // setOnInsert to avoid overwriting existing entries
+        $setOnInsert: {


Using $setOnInsert instead of $set means that existing questions will never be updated with new data from LeetCode. This could leave stale data in the database if question details change on LeetCode's side.

Suggested change

// setOnInsert to avoid overwriting existing entries

$setOnInsert: {

// set to update existing entries with latest data

$set: {

This is the intended outcome, not to update the database - to prevent issue where update may overwrite update made manually.

backend-services/leetcode-backend-service/src/db/model/question.ts

Copilot

Pull Request Overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

backend-services/leetcode-backend-service/src/leetcode/seedBatch.ts

Copilot · 2025-10-12T16:53:38Z

backend-services/leetcode-backend-service/src/leetcode/seedBatch.ts

        },
      },


The updatedAt field was removed from $setOnInsert but upsert: true still allows updates. This could lead to inconsistent timestamp behavior for existing records. Consider either adding updatedAt back or clarifying the intended behavior for record updates.

Suggested change

},

},

updatedAt: new Date(),

},

$set: {

updatedAt: new Date(),

},

In our case, an additional

$set: { updatedAt: new Date(), },

is not required since we only want to insert new records and don't want to update existing ones. Moreover, since the schema already has { timestamps: true }, MongoDB will take care of setting updatedAt when a new document is inserted.

Copilot

Pull Request Overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

backend-services/leetcode-backend-service/src/leetcode/seedBatch.ts

backend-services/leetcode-backend-service/src/health.ts

Copilot

Pull Request Overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

backend-services/leetcode-backend-service/src/leetcode/seedBatch.ts

Copilot · 2025-10-17T15:36:32Z

backend-services/leetcode-backend-service/src/leetcode/seedBatch.ts

-        },
-
-        $setOnInsert: {
          createdAt: new Date(),


The createdAt field is set using new Date() which will be the time when the batch operation runs, not when the question was originally created. Consider using a consistent timestamp or the question's actual creation date if available from the LeetCode API.

Suggested change

createdAt: new Date(),

createdAt: q.createdAt ? new Date(q.createdAt) : new Date(),

For now, we will keep to the time when the batch operation runs for tracking purposes.

Ryuse

LGTM!

Copilot AI review requested due to automatic review settings October 12, 2025 16:14

Copilot AI reviewed Oct 12, 2025

View reviewed changes

xGladiate changed the title ~~fix: Fix Question Service bugs and edge cases~~ fix: Fix Question Service seed-batch bugs and edge cases Oct 12, 2025

xGladiate self-assigned this Oct 12, 2025

Copilot AI review requested due to automatic review settings October 12, 2025 16:40

This comment was marked as resolved.

Sign in to view

Copilot AI review requested due to automatic review settings October 12, 2025 16:53

Copilot AI reviewed Oct 12, 2025

View reviewed changes

xGladiate force-pushed the fix-question-edge-cases branch from d7a163c to ceaa880 Compare October 14, 2025 15:26

Copilot AI review requested due to automatic review settings October 15, 2025 16:04

xGladiate force-pushed the fix-question-edge-cases branch from ceaa880 to c970774 Compare October 15, 2025 16:04

Copilot AI reviewed Oct 15, 2025

View reviewed changes

backend-services/leetcode-backend-service/src/leetcode/seedBatch.ts Outdated Show resolved Hide resolved

backend-services/leetcode-backend-service/src/leetcode/seedBatch.ts Show resolved Hide resolved

backend-services/leetcode-backend-service/src/health.ts Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings October 16, 2025 16:33

xGladiate marked this pull request as ready for review October 16, 2025 16:33

xGladiate requested review from Ryuse, ToxicOptimism, jiahui0309 and wzhua02 October 16, 2025 16:33

This comment was marked as resolved.

Sign in to view

xGladiate requested review from Copilot and removed request for Ryuse, ToxicOptimism, jiahui0309 and wzhua02 October 16, 2025 16:48

Copilot AI reviewed Oct 17, 2025

View reviewed changes

xGladiate added 5 commits October 17, 2025 23:36

Fix sync when retrieving all questions

6743375

Replace insertOne with updateOne but with $setOnInsert only

51b8417

Remove unnecessary unique constrain enforcement

e2dd31b

Remove initial_count from return

fd3cd12

Remove fully initial_count from fetchNonPaidQuestionList

36982ed

xGladiate added 12 commits October 17, 2025 23:36

Fix docker run command in question-backend-service README

afcdd04

Add infrastructure for failedQuestion storing

a213802

Refactor fetchNonPaidQuestionInfo to reduce repetition

169d6cd

Fix comments

c6d5e95

Add debug statement

7f2a81f

Replace magic number with constant

3d1a492

Remove quesion and dbSchema for failedQuestion

bf50547

Fix minor comment and error messages

3ed9d26

Add error handling for failed leetcode connection

93259cd

Fix README.md

9929a2d

Fix naming issues

1259069

Fix README for leetcode-service

de94690

xGladiate force-pushed the fix-question-edge-cases branch from 7ae7ebc to de94690 Compare October 17, 2025 15:36

xGladiate added 2 commits October 17, 2025 23:38

Fix magic number issue

f34d832

Remove redundant params

cd2f45f

xGladiate enabled auto-merge (squash) October 17, 2025 15:47

Copilot AI review requested due to automatic review settings October 17, 2025 15:47

This comment was marked as resolved.

Sign in to view

Fix reties count and comments for setOnInsert

43bffb3

xGladiate requested review from Ryuse, ToxicOptimism, jiahui0309 and wzhua02 October 17, 2025 15:52

Add new line for .env.example

72d9ba5

Copilot AI review requested due to automatic review settings October 17, 2025 15:56

This comment was marked as resolved.

Sign in to view

Fix retries issue

1f16df4

Ryuse approved these changes Oct 17, 2025

View reviewed changes

xGladiate merged commit fadfb16 into master Oct 17, 2025
18 checks passed

xGladiate deleted the fix-question-edge-cases branch October 17, 2025 16:02

		// setOnInsert to avoid overwriting existing entries
		$setOnInsert: {

-        },
-      },
+          updatedAt: new Date(),
+        },
+        $set: {
+          updatedAt: new Date(),
+        },

	createdAt: new Date(),
	createdAt: q.createdAt ? new Date(q.createdAt) : new Date(),

Uh oh!

fix: Fix Question Service seed-batch bugs and edge cases #35

fix: Fix Question Service seed-batch bugs and edge cases #35

Uh oh!

Conversation

xGladiate commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Changes Made

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Oct 12, 2025

Choose a reason for hiding this comment

Uh oh!

xGladiate Oct 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI Oct 12, 2025

Choose a reason for hiding this comment

Uh oh!

xGladiate Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

xGladiate Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Ryuse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xGladiate commented Oct 12, 2025 •

edited

Loading

xGladiate Oct 12, 2025 •

edited

Loading